‘I was fascinated by the process of turning a huge array of unreadable data into a source of useful information and insights for business, so I chose Business Intelligence,’ says Iryna Shkinder, Software Engineer, Data Warehouse & Business Intelligence, ЕРАМ.
Iryna graduated from the BI Laboratory in 2017 and ever since she has been combining work on the project with teaching, mentoring, and talks at thematic conferences, as well as actively participating in the adaptation of training programs for beginners in Business Intelligence.
‘I was a student at the Laboratory and, after I started working on the project, I understood what knowledge would help me to perform tasks more efficiently, comfortably, and confidently,’ Iryna remarks. ‘The current program is much more extensive and is constantly being improved to meet modern realities. Besides theoretical and practical material, during training teachers also share cases from their own experience.’ At the recent Junior's Online Conference, Iryna spoke about DWBI (Data Warehouse & Business Intelligence) and its role in today's digital world.
What is Business Intelligence?
There are many definitions of this specialization, but they can all be reduced to a common denominator, so to speak. Business Intelligence is a set of processes, tools, and technologies that help businesses make important decisions based on data analysis. Some sources equate the concepts of business analytics and business intelligence, but this is a somewhat simplistic approach.
The amount of information in the modern world is growing rapidly, and the majority of data is stored in electronic format. However, in its raw form, such data has no practical use. Processed data, on the contrary, is a valuable source of information. It can be used to increase the profitability of the enterprise, identify risks at an early stage and manage them, optimize business operations, and identify weaknesses in the company’s organizational structure. Structured data helps monitor the performance of systems and processes with KPI metrics, improve customer communication, and forecast demand.
What does the job of a DWBI engineer involve?
The work of the Business Intelligence specialist can be divided into several stages.
Everything starts with working with a data source. Data can be structured or fragmented, stored in different systems and different formats. Needless to say, it is far easier to work with structured information, which commonly comes from relational databases.
The next stage is data preparation or Back End process. The purpose of this step is to extract data from its sources and move it to storage for future use. Depending on the chosen architecture, the ETL (extract -> transform -> load) or ELT (extract -> load -> transform) approach is used. (To find out what the difference is and why it matters – join our course). The data is cleaned, standardized, transformed, validated, and stored, ready-to-go.
The final phase is Front End. At this stage, you work with previously prepared data stored in the Data Warehouse or Data Lake. Various BI tools (in the Front-End context, those are the utilities for data visualization) are used to create reports, dashboards, scorecards, as well as interactive graphs, maps, and charts. Also, you may be communicating with the customer to clarify what information they need, making the infographics, or doing any other research at the stakeholders’ request.
Is it just one specialist who does all that?
It all depends on the specifics of the project, its goals, and budget. Generally, in real projects, there are 3 to 4 main roles, and most often they are performed by different people. One engineer can combine related roles; however, one person is rarely responsible for the whole process from start to finish.
To ensure effective work and further scalability of the system, data must be carefully and correctly prepared. This is the job of a DWBI (or ETL) Developer. This specialist develops a data model, establishes the process of obtaining, transforming, and loading data into the repository, and cleansing it for further use in analytics. This person writes code, integrates all tools and processes. It is crucial for a DWBI developer to know SQL well, as it is the basis of all tools. Also, DWBI developers must have a thorough knowledge of the general principles of working with databases, because it greatly affects the quality of the code and final data. Aside from that, they need to understand the principles of building ETL processes and data warehouses and have relevant skills for that, and they should be familiar with Big Data tools (MapReduce, Hadoop, Spark, etc.) as well. For further development in the profession, you need skills in cloud computing (AWS, Google Cloud, Microsoft Azure, etc.), and knowledge of one of the high-level programming languages. Consider Python, Java, or Scala, as the most commonly used ones.
Data Quality Engineer evaluates the correctness of ETL processes’ set up, performs prototypes’ testing, writes tests and documentation, checks the product for compliance with customer requirements. Their area of expertise also encompasses knowledge of SQL and principles of working with databases, understanding principles of designing ETL processes and data warehouses. In fact, Quality Engineer should have the same theoretical basis as DWBI Developer, but with a focus on testing. In addition, you need performance testing skills, as Business Intelligence involves working with a large amount of data.
Data visualization and reporting is the job of a Business Intelligence Reporting Developer. Primarily, this specialist develops new and maintains existing reports and dashboards. Their responsibilities may also include configuring existing BI tools and improving the performance of developed reports. Business Intelligence reporting developers often work directly with stakeholders, so apart from knowing how to use specific tools for creating dashboards (and their number already exceeds dozens and is growing rapidly), SQL knowledge and experience with databases, they need good communication skills and sound command of a foreign language when working with international clients.
A Data Analyst performs a similar role. They analyze information to identify certain dependencies and trends. To properly draw logical conclusions, a specialist in this field must be well versed in the domain of the project — medicine, tourism, banking, etc. The experience of data modeling (UML, logical models) will also come in handy, as well as SQL and experience with databases.
How to start a career in Business Intelligence?
The list of basic skills for beginners is not overwhelming. The basics include SQL, understanding the key concepts of relational databases, and English. Being familiar with the algorithms and basic principles of OOP, understanding the concepts of Data Warehouse and Data Lake, having experience with relational databases (MySQL, MS SQL, Oracle, PostgreSQL, etc.), tools for data visualization and/or BI platforms (Tableau, Power BI, QlikView, etc) will be additional plusses.
However, bear in mind that the Data Warehouse and Business Intelligence online program from EPAM University is designed for those just taking their first steps in the profession and covers all the necessary topics.
Those who decide to continue their development in the profession need to be prepared to regularly and quickly learn new technologies and tools, including mastering one of the programming languages, cloud technologies, Business Intelligence platforms, tools from the BigData stack (Hadoop, Hive, Spark), ETL tools (Informatica, Talend, Pentaho), Bash scripting.
Useful links for beginners:
● Data Engineers: self-study materials
● Links to entry-level self-training materials
● Books on Relational DB (for example, books by Tom Kyte about Oracle, or by Icik Ben-Gan about MS SQL)
