Kaizen-based Firmware Development for Programmable Logic Controller
- Summary
I was responsible for implementing the motion control algorithms embedded within Programmable Logic Controller (PLC) for wide range of applications such as packing machines and filling machines. Throughout 3 years of experience under pressure from manufacturing clients all over the world, I utilized the "Kaizen" process to update the rules of the development process, actively involving all team members. This collaborative approach resulted in the release of a high-quality, high-performance product.
- Team & Role
This development was the largest project in the history of Omron Corporation. Over 100 developers had to begin implementation from scratch, as the project introduced a new platform that innovated upon the traditional ASIC-based architecture. I was responsible for implementing the motion algorithms, which are core parts of the flagship product. Developed highly optimized algorithms capable of calculating command positions for 126 motors within 1 millisecond, ensuring precise and efficient motion control.
- Results
I successfully integrated the motion core function into the products by utilizing the "Kaizen" process— a methodology focused on continuous and incremental improvements involving everyone in the organization. I expanded Japanese "Kaizen" process to an external department at the branch office in Spain, which achieved a 50% reduction in testing time.
- Challenges
During the integration phase of this project, we encountered unexpected software bugs caused by miscommunications. Under pressure from clients anticipating the innovative PLC, we had to resolve all bugs while simultaneously conducting integration tests. Despite our best efforts, the bug curve continued to rise. It felt like we were on an endless march with no clear resolution in sight.
- Solutions
To maximize the performance of all team members, I utilized the "Kaizen" process to refine the rules of development process. The concept of the "Kaizen" focuses on identifying the root causes of recurring bugs and implementing fundamental solutions with the active involvement of all team members. The essential aspect of the "Kaizen" is detecting these root causes through a structured process that examines 'When', 'Where', and the 'Why-Why-Why' analysis.
(1) Step 1: When
The first step in analyzing a bug reported by the testing team was to determine the conditions under which the unexpected behavior occurs. While some bugs arise during normal scenarios, others occur in rare cases where specific conditions accsidentally coninside. One of the most difficult errors was that a motor suddenly stopped, when the controller was operating under heavy computational load. This phenomenon was observed across several test cases. By analyzing parameters, I generalized the situation and clarified the conditions encompassing all error cases.
(2) Step 2: Where
The next step in analyzing the bug is to identify the specific function causing the issue. By referencing the software architecture, clarified the responsibilities of each component and pinpointed the function where a gap existed between the expected and actual behavior. The critical skill at this stage is a comprehensive understanding of the ideal software structure, including both software functions and hardware components. I figured out the bug described in the previous section was related to the watchdog timer and a function designed to temporarily extend the computational cycle time. This enabled me to pinpoint the specific function that failed to account for this exceptional case.
(3) Step 3: Why-Why-Why
The most critical step in the Kaizen process is investigating the root cause. Repeatedly asking "why" is an essential and unavoidable practice to reveal the fundamental issue. After thorough analysis, I realized that most of the causes stemmed from these aspects.| Categories | Factors |
|---|---|
| Human |
|
| Process |
|
This analysis enabled me to clarify both (1) the reasons for mixing the bug and (2) the causes for its missed detection. Furthermore, it helped in proposing both short-term and long-term measures. The most important aspect was that this analysis provided valuable new insights, not only for me, but also for the entire team as well. In fact, identifying the couse of this bug encouraged the team to acknowledge the importance of grasping the whole system.
One bug has changed a team's mindset.