Without a doubt, this has been an Incident that attracted a lot of attention around the world. Within the community of those who are dedicated to Industrial Cybersecurity we know that in Ukraine they dedicate a large amount of resources to the protection of Critical Infrastructure, and especially in the Energy sector, perhaps because they have already been in the news in other episodes of the past.

Since December last year, they would have been caught in a major incident. We all wonder what happened? How could such a thing have happened in a country in which a large amount of resources is dedicated to the protection of critical infrastructure? The statistics of incidents and attacks on Control Systems (IACS, Industrial Automation Control Systems) are increasing year after year, and in all parts of the world, such as in America, Europe, Asia, etc.

Both in its quantity, as in the penetration technique, and the damage that these incidents are causing to their victims. This will only be a reason for analysis in one of our next articles. In general, attacks on Critical Infrastructure generated in IT Networks or rather TCP/IP networks, very difficult to cause significant damage or have consequences on the physical world and on the population. On December 23, 2015, in the Western Part of Ukraine including the capital city of Ivano-Frankivsk, more than 700,000 residents and 200,000 industries suffered a prolonged power outage as a result of a Cyberattack.

The analysis of this case is especially valuable, to understand what was the impact of this incident that involved Control Systems and Industrial Cybersecurity as a whole. The western part of Ukraine is operated by several private energy transport and distribution companies. In this territory, the sources of Energy come mainly from Thermal Generation Plants. Ukraine also has Nuclear Power Plants, but not in this region of the country. Perhaps one might even think that the blow came from the least expected side.

The interconnected electrical system of one of the countries, with supposedly the critical infrastructure well protected, was seriously affected.

On December 23, 2015, a normal day of work in the harsh winter, at 4:00 p.m. numerous users and industries no longer had the supply of electricity. The situation began to recover from 7:00 p.m. We know that this is a long time, especially for the inhabitants and industrialists. Not only because of the number of hours, but also because of the immense number of users and a large region affected. The economic damage was enormous.

However, from the point of view of response time, and due to the magnitude of the incident, the response team responded quickly. Most energy companies in all parts of the world would not have been in a position to respond in the same way in such a short time. When the vast majority would have been totally taken aback, the response team knew how to handle the situation effectively. News of the incident began to spread throughout the world’s media.

Before the end of the year, it was already the most resonant Industrial Cybersecurity incident of the year. The research agencies of Industrial Cybersecurity were surprised and at the same time interested in collaborating with the research. What happened? And How did it happen? While the analysis and expertise continue, even to this day, several issues are already very clear.

1. BlackEnergy malware infected several computers in the IT Network. BlackEnergy (also known as DarkEnergy) is a malware that has existed since 2008 and each of its modules has been mutating over time. In this incident, the third variant of BlackEnergy was used as an attack vector and has given attackers access to the computers of power distribution companies, and the ability to communicate remotely to them. It has not been detected that this malware has reached the operating stations of the Control Systems.

However, it is certain that the intruders have managed to obtain and steal information from the Interconnected Electrical System. One of BlackEnergy’s modules, known as KillDisk, was used to deny access to corporate computers as one of several distraction and delay measures on the same day of the incident. Previous reports, they revealed, that in the month of March attempts to enter the electricity companies with the second version of BlackEnergy would have been detected. In the month of July, the alert of these attacks would have been ceased. It should have been taken as a warning and not as an action that was already controlled.

2. An attack affected at the same time the Telephony system of several of the companies. Another simultaneous attack affected the communications and telephony system of several energy companies, possibly a denial of service (DoS), prevented the help desk of the energy transport companies from receiving the complaints of the affected users. This situation is already giving clear evidence that the Incident was absolutely planned.

The power outages began quite some time before companies became aware of what was really going on. One of the providers even placed messages on their Websites informing their visitors that the company was having problems with the power supply in several locations, but they could not need an hour of restoration. The situation was critical, and they didn’t know what was happening. The blows came from all sides.

3. Several of the Sub-station Control Systems were compromised. When the response teams went to re-establish the control systems of each of the power sub-stations, they realized that the Field Control Systems, PLCs and RTUs, had also been compromised. Subsequently, it was found that the firmware of the devices were corrupted or modified, and their configurations had been altered. The control systems of 16 sub-stations were found in these conditions. Another clear evidence that the incident occurred was no coincidence. The electric power distribution companies of western Ukraine had been victims of an intentional attack, with high motivation and advanced technical knowledge of Control Systems.

4. The Operating Stations of some SCADA Systems were accessed remotely. An operator, minutes before making his shift change, recounted how suddenly a ghost took control of his workstation, and began to act on the sub-stations. In just a few seconds, he witnessed how the supposed ghost just left a large number of users and industries without power. This ghost was able to circumvent verification and confirmation processes of the operator’s actions.

The Operator had no time or possibility to do anything to prevent it. It wasn’t a ghost! Remote access through VPN links had been breached. They were operating the SCADA from the outside, and the Hackers were in control of the system. The operated could no longer see what was happening. Their Access Keys did not work. The same security measures of the system itself had blocked the Operators from being able to access. A Control System operator located in Prykarpattyaoblenergo, while he was performing his paperwork to finish his work shift, suddenly observed how the cursor of his System Operation Station, suddenly shifted towards a corner of the screen. Quickly, the cursor began to perform a series of actions on the screen and maneuvers in the sub-stations.

The ghost operator selected a substation, went directly to it without wasting time, knowing exactly where it should itch, selected the substation, gave the order to shut down. The SCADA System has a confirmation sequence, as is typical for SCADA systems of the Electric type (Select Before Operate, and others), gave the order to shut down, confirmed the sequence, as an expert and so on with the 30 sub-stations of that operator.

The one-line diagrams and different screens are not easy to understand, and you must have a knowledge of the Electrical Networks. The operator tried to stop this process, but could do nothing. The worst had already begun, even before this incident. Numbers of users were suffering the consequences of power outages even before the operator incident. This was not all. At the same time, similar incidents were happening in other control centers of other energy companies. The number of sub-stations that were being put out of service was multiplying rapidly.

A little more than three months after the incident occurred and after knowing only some of the results of the investigations that were known, through the different investigation agencies, there is no doubt that it was a Cyberattack … planned, elaborated and executed with great effectiveness. Perhaps those responsible would think that the damage they would cause would be much greater. The efficient response in the Recovery of the Incident by the Response team made the losses not even much greater. What would have happened if the incident continued throughout the night? Those responsible were not opportunists. They knew perfectly well what they were doing, and they had it absolutely premeditated.

They were strategic, detail-oriented and planned for months. Multiple attacks orchestrated with synchronize as an authentic choreography. When most people hold malware accountable, it is clear that behind this incident there was significant logistics and motivation. Given these facts, the malware could have been a distraction to hide the real vectors of the attack. In fact, no malware was found in the control systems or in the SCADA Operation Stations. The protections installed on the systems to prevent entry from the corporate network have proven to be effective.

However, the activist group faced with this situation did not give up and looked for other ways to enter the control systems. The different perspectives of the attacks. When some believe that all these damages were caused by a hacker or a group of hackers from IT Networks, we know that reaching the low level of control, this is not possible. Changes to firmware and in the configuration and programming of the Control Systems are not possible to make simply by accessing the Operating Room or the SCADA Room, as one of the operators related, or through the BlackEnergy Malware found in the IT Networks.

Especially in an Electric SCADA with networks of several years, where to access the firmware of field devices, make modifications without this arousing suspicion without generating events, It is not possible. There is no doubt that there was an intelligence task, there was planning, and even more, there was a knowledge not only of the Electricity Grid of the region but also of the SCADA, PLCs, RTUs, communications and their vulnerabilities. It is also known that the loss of the SCADA system, in a well-conceived system, does not generate the fall in the supply of energy. To do this, PLCs and RTUs should be reached.

While technically feasible under laboratory conditions, it is highly unlikely if not impossible that the control systems in the substations have been altered by the use of BlackEnergy Malware. To make the changes found in the PLCs and field RTUs in the Sub-stations requires direct access and in many cases very specific knowledge, and specific software. Moreover, these changes are not possible to make in a few minutes, it takes many hours to alter the Control Systems similarly and even more to go unnoticed.

Changes to firmware and configuration are not the same as making changes to the SCADA monitoring and operation level than to the control level. We believe that other incidents and events not yet disclosed took place. It is reasonable that confusion is generated by lack of knowledge of the global community. In fact, in most of the incidents that have caused real damage to physical infrastructure, the capacity of the attackers has been underestimated, as was the case with the German Steel Industry, assuming that the access was made from the Corporate Network or from the Internet, which has been false in 90% of the incidents, or simply from a VPN access. The intervention in the Telephony System was not the cause of the massive blackouts, either.

This served to delay the response team in detecting and becoming aware of the problem. This was also a distraction. A report by the Energy Industry Research Center describes Ukraine’s SCADA systems as technologically outdated and should be revamped. Therefore, getting remote access to PLCs and RTUs makes it even more unlikely, if not impossible. Even more so when no evidence of BlackEnergy 3 was found in the Control systems. If it was the case with Shamoon and Dragonfly in other incidents that happened in the past. However, this time we are facing a unique fact. PLCs and RTUs were entered and modifications were made to them. Even firmware changes. How is it possible that a “power” in Cybersecurity is a victim of such an attack? How is it possible that he was so vulnerable? What have you been doing all this time to protect your control systems? From Ukraine, they quickly pointed to the Russians as responsible for the attack.

However, due to the type and level of sophistication of the work carried out, the incident must have been caused by several actors and/or participants. They could even have achieved the collaboration of completely disconnected or disinterested parties who for a few dollars are able to do anything, without measuring the consequences. The United States of America recognizes that Ukraine’s Control Systems are more secure than its own. However, for the attackers, it seemed like an easy job. Failures in security measures. SCADA System’s VPN remote access had a single authentication factor, when it should have a minimum of two.

The blackout took a total of 6 hours. However, even after three months of what happened, the control systems are not fully operational. Several of the sub-stations continue to be operated manually. Restoring control systems where nothing can be underestimated is no longer easy. It is necessary to recover, test and verify the system at a level of detail takes weeks. They can’t even rely on backups. Still, they can’t leave the systems as they were before the incident. You want to change everything!!! Otherwise, the same thing will happen again.

First Phase: During the surveys, no evidence was found that the BlackEnergy3 Malware had reached the control systems. On the contrary, everything seems to indicate that the protection measures to prevent intrusions from the corporate area to the control systems were effective. The attacks would have started several months earlier on IT networks. The attackers did not give up and looked for other ways to enter the control systems. These attacks in IT areas were executed through the sending of emails, with Word documents asking for permission to run macros. These macros would have the malware. Even gaining access to the computers of the administrator profiles, they did not manage to reach the control systems. The different firewalls and logs showed that all entry attempts were blocked. What was missing was a proper analysis in time. The attackers had two options. Look for vulnerabilities in firewalls and barriers to entry to control systems from the IT networks, or look for alternative ways of entry. The attackers eventually opted for the second option. They should gain one or more direct access to the control systems of the multiple energy distribution companies.

Phase Two: Attackers began looking for alternative forms of entry. They managed to enter the domain server and stole the login credentials through the VPNs of various SKAs. Once they managed to enter the SCADA System Network, they began to weave their strategy. They reconfigured the UPS into two of the Control Systems. In addition to leaving the inhabitants without energy, they managed to blind the operators, but they did not manage to make this move in the other systems. In some systems, they simply managed to free the screens of the operators. It was enough that they couldn’t see what was going on in the field. The technological differences of the different control systems, were providing different income opportunities. The companies used different DMS (Distribution Management Systems). They managed to modify the firmware in various equipment and network communications devices. In addition, they changed the firmware in several Control Systems, at least in a minimum of 16 sub-stations. It is the first time that such a case is detected, where the actors manage to make a firmware change in the Control Systems. From the perspective of an attack, it was all engineering. Once all these changes were achieved, they were ready to execute their attack.

Third Phase: once a sufficient number of attack vectors and Trojan horses were deployed, what they had to do for the attack to be effective and have a great impact, was to act in a coordinated manner. And so they did. The electricity supply service was restored almost entirely manually. The response team observed that it was the only way they had to recover power service. (It wasn’t known what was going on until after several weeks and months of analyzing the systems.) They had to do it manually without the help of control systems. In this case it was possible, however, in the most modern electrical networks this would not be possible and to restore the service they should recover the control systems first.

We know that in Ukraine they invest a lot of resources to protect their Critical infrastructure. However, even being a country prepared in terms of Cybersecurity, they have been victims of one of the most feared attacks. This also shows that the defense strategy must be implemented in multiple layers. It is not enough to protect a single entrance. This is like pretending to protect our home by guarding only the service door.

For an Industrial Cyber Defense system to be effective, both Comprehensive Detection and Comprehensive Protection must be implemented. All doors, all windows, up to the entrance of the dog or cat, the fireplace and the garage should be monitored and protected. It is not enough to protect TCP/IP Data Networks, as is being done in most cases. A Comprehensive Cyber Defense system must be implemented. Otherwise, all the effort may be in vain.

Let’s be alert as more information is being revealed. As with the Stuxnet case, it took a while to know everything that really happened. This will be no exception.

Lesson #1: The attackers required a minimum of 6 months of reconnaissance, research, and orchestration to carry out their attack. With the right tools, and there are free to use, the activities of this group would have been detected.

Lesson #2: Remote connections of SCADA Systems and/or Control Systems must have at least two authentication factors.

Lesson #3: The Power Supplies and UPS of Control Systems are as critical as the Control System itself. These must be safe and reliable.

Lesson #4: Control System Firmware Updates are also a valid and very critical attack vector, which most community actors do not yet have in mind.

Lesson #5: A Comprehensive Detection System as well as Comprehensive Protection should be implemented. It is not enough to protect only some of the entry points.

Lesson #6: A comprehensive study of vulnerability identification and risk assessment to Industrial Cybersecurity should be conducted before implementing protection measures. Failure to do so runs a high risk of underestimating attackers. Whether intentional or unintentional.

Lesson #7: All the information, documentation and details that describe the operation of the Electric Grid and the architecture of the Control Systems must be secured. It can provide attacking stakeholders with the information needed to identify vulnerabilities in Control Systems.

When was the last time you conducted a review of your Incident Response Plan at your plant or processes in your company? Do you have a response plan? Are your people mentally and psychologically prepared to respond to these types of incidents? Or would you be totally caught off guard with your pants down? Could your organization respond to something similar? Have you completed an Industrial Cybersecurity Risk Assessment? Are you aware of what may happen in your processes? What is the current state of your Cyber Defenses? Are you following any methodology? Which one? Is it the right one?

In short, even if you are not in the electrical industry, this incident is a very good opportunity to observe, learn and analyze what your current situation is. For all those providers of Critical Process Systems and Services, it is vital to learn from these cases.

Would you like to leave your opinions and comments down here at this location? Write to us at support@wiseplant.com