The amount of information available to companies today is gigantic, and every day, even every minute, depending on the industry, new data is being generated. Big data brings several challenges for companies since the amount of space and processing capacity increases as the data grows, and many times companies and employees are not aware of or do not know the implications of the volume being handled.
Big data challenges are related in the following aspects:
– Collection: The first step in having data is to collect the data. The collection includes the form or medium from where I will get it. It is necessary to decide all the information being generated that I need to take, all the data, only some, are data that are already structured or are data like images, videos, texts, etc.
– Storage: The second step is how and where I will organize it. If you collect data in real time or very often, the data size will grow exponentially. At this point, decisions must be made about how long to store the information, what type of database and services I will use to store it, whether I will use file compression techniques, and the elimination of duplicates, among other things. Also, during this stage, information security and access permissions must be considered.
– Processing: Big data processing involves selecting the technological tools to be used to analyze and store the information. Which tool available in the market is the best or the one that best meets the specific needs of the Enterprise are decisions to be made by the company’s technical team or by using consultants who can guide the selection of the best tool.
Finally, there is a human component within Big data since it is the engineers and analysts who are in charge of making the above decisions, and they are the ones who have to use this data to generate value for the company, otherwise, it should not be storing the information.
The human departments have a huge challenge in finding suitable profiles and training programs to meet these needs. Among the profiles needed for these projects are data scientists, data analysts, and data engineers.
References
Laura, Nice Post! Interesting idea of how big data effects organizations. I agree big data can certainly be a problem when it comes to processing and analyzing such data to generate meaningful and actionable outcomes to support business strategies. I know there are tools that exist to support such big data, utilizing AI/machine learning to help analysis and categorize data for further analysis. How do we manage the data in a way that can support data-driven business architecture? What tools exist? How can we mitigate those big data challenges?
Hi Laura,
What a great post! I think you did a great job summarizing some of the most important considerations that enterprises must make surrounding big data. I only note that within your mention of the “human component” you didn’t mention the security of the data, nor the ethics of collecting and using the data.
Very certainly rules about data and how it can be used differ geographically, but considering some of the big international scandals we’ve seen surrounding the less-than-ethical use of data (Cambridge Analytica for example) what are your feelings about the cross-cutting component of security in EA as it concerns data security and privacy? Considering that many large enterprises place value in the form of share price over the ethical implications of their activities what do you think is the best way to safeguard the data of folks who may not even know that their data is being collected and used?