Luyện English nhờ chúng m giúp thằng em họ t thoát khỏi đặc khu
- Tạo bởi LQDuy
- Start date
Aws glue is a f*cking scam
I have been using aws glue in my project, not because I like but because my previous team lead was a everything aws tool type of guy. You know one who is too obsessed with aws. Yeah that kind of guy.
Not only I was force to use it but he told to only use visual editor of it. Yeah you guess it right, visual editor. So nothing can be handle code wise. Not only that, he also even try to stop me for usings query block. You know how in informatica, there is different type of nodes for join, left join, union, group by. It similar in glue.yeah he wanted me to use it.
That not it, our pipe line is for a portal which have large use base which need data before business hours. So it's need to effecient an there is genuine loss if we miss SLA.
Now let's talk about what wrong with aws glue. It provide another python class layer called awsglue. They claim this layer optimize our operation on dataframe, in conclusion faster jobs.
They are LIARS. There is no way to bulck insert in mysql using only this aws layer. And i have tested it in comparison to vanilla pyspark and it's much slower for huge amount of data. It's seems they want it to be slow so they earn more money.
Dimensional Modeling Question
I work at a large manufacturing company as a business analyst and I'm trying to use some of my real world experience as way to help me understand data engineering and data warehousing concepts for a potential career change, either within or outside my current company. Anyway, I'm very familiar with transactional databases and highly normalized schemas and it's proving to be bit difficult to shift my mindset to a denormalized star schema. One of the things that tripping me up is related to 1-to-many relationships in dimensional modelling.
I understand that it's ideal to flatten the data as much as possible, but some things simply don't work that way. For example, we make products from chemicals and each product can have an indeterminate number of chemicals that make up their "chemical composition." Sometimes there is a need to report something along the lines of "for each product, provide the chemical composition" while others are more along the lines of, "these chemicals are now regulated differently than before, find all products containing them" or "figure out how many tons of a given chemical were imported into the country." In the first example, I could see a product dimension containing maybe a JSON column representing the composition attributes, but this seems like it would create a nightmare for performance in the second and third example.
The obvious solution is to just normalize it like our transactional system where there is a product table and a composition table, where each chemical is linked to the product by a foreign key. This is slightly denormalized relative to our transactional database where there is a product, chemical, chemical regulations, product composition table, but it still doesn't seem to align well with the star schema. I've been reading The Data Warehouse Toolkit and trying to find examples that closely match the reporting required in this industry and it's just hard to find anything really similar that is covered.
Thanks in advance for the help!
I work at a large manufacturing company as a business analyst and I'm trying to use some of my real world experience as way to help me understand data engineering and data warehousing concepts for a potential career change, either within or outside my current company. Anyway, I'm very familiar with transactional databases and highly normalized schemas and it's proving to be bit difficult to shift my mindset to a denormalized star schema. One of the things that tripping me up is related to 1-to-many relationships in dimensional modelling.
I understand that it's ideal to flatten the data as much as possible, but some things simply don't work that way. For example, we make products from chemicals and each product can have an indeterminate number of chemicals that make up their "chemical composition." Sometimes there is a need to report something along the lines of "for each product, provide the chemical composition" while others are more along the lines of, "these chemicals are now regulated differently than before, find all products containing them" or "figure out how many tons of a given chemical were imported into the country." In the first example, I could see a product dimension containing maybe a JSON column representing the composition attributes, but this seems like it would create a nightmare for performance in the second and third example.
The obvious solution is to just normalize it like our transactional system where there is a product table and a composition table, where each chemical is linked to the product by a foreign key. This is slightly denormalized relative to our transactional database where there is a product, chemical, chemical regulations, product composition table, but it still doesn't seem to align well with the star schema. I've been reading The Data Warehouse Toolkit and trying to find examples that closely match the reporting required in this industry and it's just hard to find anything really similar that is covered.
Thanks in advance for the help!
Bro learned something that day
What a dude. Took the situation for what it was and thanked the dude for saving him time and energy. Didnt even bother acknowledging her existence. She so deserves that. He would have found out later somehow anyhow and he realized. What an intelligent drunk man.
I was that dude once. I was with girl I though we have something going on, but she refused to call us a couple in front of people, called me cute names only in private ... so we had discussion and she acted "shocked" that I thought we are in relationship. Funny thing is that she completely lost her shit when I found somebody else. She basicaly thrashed my place. Yeah, valuable exp.
but she didn't say she wasn't in a relationship?
What a dude. Took the situation for what it was and thanked the dude for saving him time and energy. Didnt even bother acknowledging her existence. She so deserves that. He would have found out later somehow anyhow and he realized. What an intelligent drunk man.
I was that dude once. I was with girl I though we have something going on, but she refused to call us a couple in front of people, called me cute names only in private ... so we had discussion and she acted "shocked" that I thought we are in relationship. Funny thing is that she completely lost her shit when I found somebody else. She basicaly thrashed my place. Yeah, valuable exp.
but she didn't say she wasn't in a relationship?
how is it possible for this man to have this haircut 
this is like a cartoon character google his name he always has this haircut
looks like it is his first and last haircut
looks like said YES to a "become a billionaire but you always have to rock this haircut" deal from some magical fish
this is like a cartoon character google his name he always has this haircut looks like it is his first and last haircut
looks like said YES to a "become a billionaire but you always have to rock this haircut" deal from some magical fish
Why Power BI
Why is Power BI suddenly being implemented in every company, FMCG sector, Insurance and financial institutions.
Is it because of their cheap licensing strategy?Being part of Microsoft Ecosystem?
Can it be used for quick and dirty or serious analytics?
SAS and others are so expensive it becomes for the analytics team to justify.
Backdrop:
Analytics teams are no more decision making centers on Budget unless it comes from top
Why is Power BI suddenly being implemented in every company, FMCG sector, Insurance and financial institutions.
Is it because of their cheap licensing strategy?Being part of Microsoft Ecosystem?
Can it be used for quick and dirty or serious analytics?
SAS and others are so expensive it becomes for the analytics team to justify.
Backdrop:
Analytics teams are no more decision making centers on Budget unless it comes from top
Finished my CS degree and know nothing about programming.
Im 22 , finished uni at 21 and have absolutely no idea what i am doing, the past year has been spent mostly gaming and procrastinating, im interested in javascript i think. Any advice , and is it too late to start over on learning how to code ?? Also i think web programming suits me best, i spent my 3 years of uni slacking off due to personal and family issues , this feels like a useless vent post but i really feel directionless and pressured to secure an internship.
Im 22 , finished uni at 21 and have absolutely no idea what i am doing, the past year has been spent mostly gaming and procrastinating, im interested in javascript i think. Any advice , and is it too late to start over on learning how to code ?? Also i think web programming suits me best, i spent my 3 years of uni slacking off due to personal and family issues , this feels like a useless vent post but i really feel directionless and pressured to secure an internship.
Ask Devops: How do you differentiate dev, test, and prod?
Having a good environment setup that lets you test your code in prod like environments, keeps actual prod safe, and is flexible enough to ship fast is key.
We’re building one option for extremely lightweight dev environments at [Kardinal.dev](https://kardinal.dev/?&utm_medium=ppc&utm_source=reddit&utm_campaign=usdevops&utm_content=4_1), but we'd like to learn how our team define development, testing, staging, and production environments
Check out our github repo [HERE](https://github.com/kurtosis-tech/kardinal?&utm_medium=ppc&utm_source=reddit&utm_campaign=usdevops&utm_content=4_1)
Having a good environment setup that lets you test your code in prod like environments, keeps actual prod safe, and is flexible enough to ship fast is key.
We’re building one option for extremely lightweight dev environments at [Kardinal.dev](https://kardinal.dev/?&utm_medium=ppc&utm_source=reddit&utm_campaign=usdevops&utm_content=4_1), but we'd like to learn how our team define development, testing, staging, and production environments
Check out our github repo [HERE](https://github.com/kurtosis-tech/kardinal?&utm_medium=ppc&utm_source=reddit&utm_campaign=usdevops&utm_content=4_1)
CEO wants everyone to use an AI. I have zero idea on what I can use it for.
Ok, so I'm a linux sysadmin with \~15y of professional experience and CEO just sent a mail to everyone encouraging us to use AI because he can code 2x faster.
What do I do ? Is there anything I can use ?
Context :
- CEO is not some finance-focused guy that never touch a computer. He is a backend dev and coded all the core of the SaaS my company runs. He still codes when he is not with prospects or client, etc. so he is not out of touch with the overall tech. He just discovered that he can delegate all the coding of its unit tests to some vscode plugin interfacing him with chatgpt. So now "AI = 2x faster coding". So now every dev must use AI.
- He wants the company to pay for the AI fees if we have any because he strongly encourages (quasi-mandates) all the dev to use it. And the infra guys also.
- with my coworker, we manage \~100 linux VMs and some pfsense. That's all the infra supporting the SaaS and all the internal tooling, plus all the IT (\~80% mac, \~15% linux, 1 windows for the "cyberexpert" guy), you get the picture.
- we use ansible when we can, but not everything is that simple (no IaaS, no cloud, and a bunch rules like no access to prod with a PC that has access to internet, prod has no access outside, prod has no access to our internal git, this kind of rules because ISO12345 (not really ISO, but a state provided qualification with the same level of constraints)), so we still do a lot of manual thing in a linux term with docs and procedures.
So, do you have any ideas on what I can do ? Do you yourself use an AI (chatgpt, other ?) to manage your infra ?
Ok, so I'm a linux sysadmin with \~15y of professional experience and CEO just sent a mail to everyone encouraging us to use AI because he can code 2x faster.
What do I do ? Is there anything I can use ?
Context :
- CEO is not some finance-focused guy that never touch a computer. He is a backend dev and coded all the core of the SaaS my company runs. He still codes when he is not with prospects or client, etc. so he is not out of touch with the overall tech. He just discovered that he can delegate all the coding of its unit tests to some vscode plugin interfacing him with chatgpt. So now "AI = 2x faster coding". So now every dev must use AI.
- He wants the company to pay for the AI fees if we have any because he strongly encourages (quasi-mandates) all the dev to use it. And the infra guys also.
- with my coworker, we manage \~100 linux VMs and some pfsense. That's all the infra supporting the SaaS and all the internal tooling, plus all the IT (\~80% mac, \~15% linux, 1 windows for the "cyberexpert" guy), you get the picture.
- we use ansible when we can, but not everything is that simple (no IaaS, no cloud, and a bunch rules like no access to prod with a PC that has access to internet, prod has no access outside, prod has no access to our internal git, this kind of rules because ISO12345 (not really ISO, but a state provided qualification with the same level of constraints)), so we still do a lot of manual thing in a linux term with docs and procedures.
So, do you have any ideas on what I can do ? Do you yourself use an AI (chatgpt, other ?) to manage your infra ?
@Olineasdf kieeru như thế này a ạ.
Hello! I'm an EFL teacher who has recently started working with a data engineer, and I need to teach him how to talk about his job in English. The problem is, even though I've learned the basic terms related to this area, I'm not sure how to use them correctly in a sentence. For example, pipelines. What do you do with them? I've seen the collocation "build pipelines", but I'm sure there are much more.
So, what I'm asking here is to help me find as many of these common collocations necessary to describe your job as possible. As if you were answering "What are your job responsibilities" question very thoroughly.
Thank you!
So, what I'm asking here is to help me find as many of these common collocations necessary to describe your job as possible. As if you were answering "What are your job responsibilities" question very thoroughly.
Thank you!
How to make 10TB of data available on-demand to my users
HelpHi, I have very little knowledge of software engineering but have been working my way by learning from reddit/stack-overflow and experimentation. I would like to learn about cheap ways I could make about 10 TB of data available to my users.
As of now, I have about 1TB of data stored in my external SSD. I have attached this SSD to my wifi router and have configured NAT to make it available over the internet. It is an FTP server and I have a python wrapper that facilitates the read/write operations. This costs me about $50 per month for internet connection and a one time cost for the SSD.
For the sake of simplicity, lets assume with 10TB of data:
- Each file size is ~100 Mb
- 60000 reads per day
- 10000 writes per day
- data is partitioned by group_1/sub_group_1/sub_sub_group_1/year/month/day
I am tempted to buy more SSDs and configure them with my router. I believe with increasing requests the router with clog up and give rise to increased latency issues. I was wondering if I can get more than 1 internet connection. This way the cost of external SSDs are a 1-time cost, internet connection cost is much lower than AWS s3 and read/write is free.
Am I going in a completely wrong direction?? What are other alternate low-cost, low-latency options?
Any help/feedback direction is appreciated.
Thanks!
@npham_1309 Like cho thằng em tôi thoát khỏi đặc khu với cảm ơn bro. Thằng này @LQDuy2
Cloud compute for on prem DW
HelpMy company is looking for a new ETL process but wants to keep the existing EDW on prem SQL server. What are some solutions where we can pull data from db2, do ETL using cloud compute, and then bring it to our sql server. I thought of using an orchestration tool like airflow or dragster. Spinning up compute containers using docker. And using python/sql to script. But they also want a managed solution and something with support. Thoughts?
FYI: we are moving away from informatica and I would prefer if we don’t go with another outdated GUI tool.
chucmung122
Hạt giống tầm thần
luyện xong đi mẹ luôn cả họ 
npham_1309
Trai thôn
Chấm
npham_1309
Trai thôn
➕ Có thể bạn quan tâm
Có Hình
Nhờ ae giúp
